Make live view usable on touch devices by masnwilliams · Pull Request #448 · kernel/kernel-images

masnwilliams · 2026-10-09T20:55:12Z

Summary

Live view is hard to use from a phone or tablet: a one-finger swipe is sent as a mouse drag (it selects text instead of scrolling), a desktop-sized stream is too small to read or hit, and the soft keyboard only opens from a small icon. This PR adds a touch layer to the live view client. It is client-only: the remote browser still receives the same mouse, wheel and key messages it does today.

Changes

Gestures (src/utils/touch-gestures.ts)

Tap clicks at the tapped point.
One-finger swipe scrolls through the existing wheel message, so content follows the finger, with client-side momentum after a fling.
Long-press then release right-clicks; long-press then move drags (text selection, sliders, drag and drop).
Whether the page takes a touch is decided when it begins. A touch that begins without input, for example before control is granted, only zooms. If input stops during a touch, the page's part of it ends, including a fling.
Two-finger pinch and pan zoom the video locally (src/utils/zoom-pan.ts), up to 3x the fitted size, or 1.5x the stream's native pixel size for large streams (at most 8x). The zoomed video can fill the letterbox bars, but not the part an overlaid soft keyboard covers. A "2.5× · Reset" chip shows while zoomed and resets it. Double-tap does not reset the zoom, so two quick taps while zoomed are two clicks. Pointer coordinates are mapped through the zoom, so taps land on the right remote pixel.

Keyboard

A tap can raise the soft keyboard. The signal is the remote cursor shape: neko already sends cursor images over the data channel, and Chromium shows a text (I-beam) cursor over editable fields. The pointer moves to the touch point on touchstart.
- iOS only opens the keyboard for focus() inside the user gesture, and over a real network the cursor update usually arrives after the finger lifts. So on iOS, a tap whose cursor is not known yet focuses the overlay inside touchend with inputmode="none", which shows no keyboard. When a text cursor arrives, inputmode is removed, and WebKit then shows the keyboard for the element that is already focused (WebKit's fast/forms/ios/inputmode-removing-none.html layout test covers this). If no text cursor arrives, the overlay is blurred when the window ends. Other browsers allow a later focus() after a user gesture, so they focus when the cursor arrives.
- Pages often show another cursor briefly before the I-beam after a click, so any text cursor within 800ms of the tap counts. Only the end of that window means the tap was not on text.
- A tap under an I-beam that is already showing produces no cursor update, so a text cursor still showing at the end of the window raises the keyboard.
- The client remembers where a text cursor was seen. A tap within 12 screen px of that spot raises the keyboard inside the tap. If the cursor then turns out not to be text, the keyboard closes at the end of the window.
- Nothing appears on the page when the guess misses. A text cursor that arrives after the window, or a field whose I-beam could have been the previous tap's, is left to the keyboard button.
- A new touch, such as a scroll, pinch or drag, cancels the wait for the previous tap's cursor, so the keyboard does not open in the middle of it.
- After Enter with the keyboard open on an I-beam, the keyboard closes if the cursor under the last tap is no longer an I-beam a second later (a submitted form or a new page). A chat box or textarea keeps its I-beam, so the keyboard stays open.
- While the keyboard is open the video keeps its size. A frame that only gets shorter is the keyboard taking room, so the video moves up to keep the last tap just above the keyboard, and refits when the keyboard closes (src/utils/keyboard-pan.ts).
- Tapping a non-text area closes the keyboard. See src/utils/cursor-shape.ts for the cursor classification and src/utils/tap-keyboard.ts for the decisions.
Text from keyboards that report keyCode 229 (Android) arrives only as input and composition events, which the Guacamole keyboard ignores. These are now typed, including composition updates and backspace.
Upper-case letters from soft keyboards were typed in lower case, because no Shift press is reported and the X server maps the keysym to the unshifted key. Shift is now sent around them on touch devices. Soft keyboards report their own Shift and Caps Lock as separate key events. On iOS these are synthetic modifier events, and the Guacamole keyboard releases every iOS key right after pressing it. So on touch devices Caps Lock is never sent, the remote's Caps Lock is turned off when the keyboard opens, and Shift goes around a capital unless the remote already holds Shift (tracked from the keys sent). Key events and input-only text go through one path (TextInput in src/utils/text-input.ts). An input event that follows a key the Guacamole keyboard already sent is skipped until that key's native keyup, and so is the repeat some keyboards send after compositionend.
Extra keys: while the soft keyboard is open, a row above it has Esc, Tab, Ctrl, the four arrows and Paste (src/components/touch-keys.vue). Ctrl is one-shot: it stays visibly latched until the next key, from the row or the soft keyboard, and goes with that key only. The row's keys act on touchend and never take focus, so the keyboard stays open.
Copy and Paste: Paste is the last key of the row. Copy shows on the keyboard control for 8 seconds after a long-press drag, which usually selects text, and the next tap withdraws it.
- Paste reads the phone clipboard inside the tap (navigator.clipboard.readText, which iOS confirms with its own Paste bubble). Text up to 32 characters on one line is typed, which also works in fields that block pasting and in code fields with one box per character. Longer text is pasted the way neko's desktop path does it: it is set as the remote clipboard (control/clipboard, the only clipboard message neko's legacy protocol has), and Ctrl+V is sent once neko reports the new clipboard back. If that report does not come or holds other text, the text is typed instead. A native paste on a touch device goes the same way.
- Copy starts navigator.clipboard.write([new ClipboardItem({ 'text/plain': promise })]) inside the tap, sends Ctrl+C, and fills the promise from the next remote clipboard update over neko's clipboard sync, so iOS's user-gesture rule holds. The automatic clipboard sync skips its own write while a Copy write is pending.
- When the phone clipboard cannot be read or written (a refused permission, an iframe without clipboard permissions, a browser without the async clipboard API), neko's clipboard panel opens at the top of the view instead. It mirrors the remote clipboard, so copied text can be selected and copied natively, and text pasted into it goes to the page with "Paste into page". On desktop the panel is unchanged and still has no menu button.
- A short notice reports "Copied", "Nothing was copied" and an empty clipboard.
The 30px icon is replaced by a keyboard control (src/components/touch-controls.vue) that stays off the stream whenever it can:
- Bottom band: when the stream leaves at least a 52px band below it (a 44px target plus margins and the safe-area inset), the control sits there, labelled "Keyboard" / "Hide keyboard".
- Side strip: when the spare room is beside the stream (for example a portrait stream on a shorter screen), the control sits in a 52px strip on the left or right.
- Shrinking: when neither band is big enough, the video is shrunk by up to 10% to make one. A stream that exactly fills its frame needs 6–8% at phone sizes (52px of 664–844px). Beyond 10% the stream gets noticeably smaller, so the control overlays the stream instead.
- Overlay handle: a 44px handle docked to the left or right edge, safe-area aware, starting on the left at mid-height. Released after a drag, it snaps to the nearest edge.
- In every mode: the control can be dragged along its band, strip or edge, and dragging never sends input to the page. It fades to 30% after 3s without a tap or drag and wakes on any touch on the view. Over the stream, the first touch on a faded handle only wakes it and never reaches the page. A mouse click on the control does not blur the keyboard before toggling it. The position is stored in localStorage, the control moves up above an overlaid soft keyboard (using visualViewport), and it shows on any touch device rather than only below 768px or with ?embed.
The overlay textarea gets autocapitalize and autocorrect off, autocomplete="one-time-code" on touch devices (off on desktop) so iOS can offer a code it received, and a 16px font so iOS does not zoom the page when it focuses.

Layout

.neko-main no longer has min-width: 360px, which pushed the controls off-screen in narrow embeds.
Heights use 100dvh, so the bottom controls are not hidden under mobile browser toolbars.
viewport-fit=cover makes the safe-area insets apply.

URL option

?tapKeyboard=auto (default): raise the keyboard when the tap lands on a text cursor.
always: raise it on every tap.
off: keyboard button only.

Desktop mouse and keyboard handling is unchanged. The new input paths only run for touch events, or on touch devices (Shift wrapping, input handling and reading cursor images). The touch-only branch in focusOverlay is removed because touch no longer synthesizes mouse events.

Testing

bun test tests: 153 pass, including tests/touch-input.test.ts (gesture classification, fling and its cancellation, a cancelled drag releasing its button, long-press right-click and drag, touchcancel, touches the page does not take, zoom math and clamping, keysym mapping, soft keyboard text with key-event and composition dedupe, and an iOS soft keyboard replay through the real Guacamole keyboard covering a Shift tap, soft Caps Lock on and off, and every combination of modifier flags on the letter events), tests/touch-controls.test.ts (band, strip, shrink and overlay layout; drag and snap; stored position; a faded handle's first tap), tests/tap-keyboard.test.ts (tap keyboard decisions in every mode, driven through the state machine with a fake clock) and tests/video-input.test.ts (two taps while zoomed, ending a touch when input stops). Lint shows no new warnings compared with main, and npm run build succeeds.
Hot-patched the built client onto a running headful browser (390x844, with and without kiosk mode, plus a 1920x1080 session) and drove it from Chromium with iPhone 13 device emulation, using trusted touch input over CDP. Checked on the remote page:
- A 300px swipe scrolled about 380 remote px, matching the finger at that scale, and a fling continued about 690px.
- Tap clicked the mapped pixel, and long-press produced a contextmenu.
- Pinch to 2.5x and 4x worked, and taps while zoomed hit the exact remote pixel. At 3x, two taps 120ms apart produced two clicks on the remote page and kept the zoom (before, the first tap clicked and the second reset the zoom).
- Tapping an input raised the keyboard inside the tap on 40–160ms taps. Tapping a non-text area closed it.
- Keyboard timing, on a 1920x1080 session at 390x664 with iPhone 13 emulation, 60–120ms taps, and 150ms added to every data channel message to stand in for a phone network. Before this change, every tap on a textarea showed the "Tap to type" button and no keyboard: the first tap at about 170ms, and taps under an I-beam that was already showing at about 900ms. Typed text went nowhere. After it, the overlay was focused inside touchend with inputmode="none", and inputmode was removed when the I-beam arrived (about 170ms) or at the end of the window (about 900ms) for a tap under an existing I-beam. A tap within the remembered spot focused inside the tap. Text typed afterwards reached the remote textarea, and tapping the blank area blurred the overlay with no button. With 1000ms added, the button appeared at about 1020ms. With Pixel 7 emulation the overlay was focused when the I-beam arrived instead of inside the tap. A swipe started 40ms after a tap on the field, before the I-beam arrived, released the held focus at its touchstart and showed neither the keyboard nor the button. Before this change, the button appeared during the swipe.
- Text typed through input events, IME composition with a keyCode 229 backspace, and key events with capitals all arrived correctly.
- With the browser UI visible, tapping the address bar raised the keyboard and typing a URL plus Enter navigated.
- Keyboard control layout:
  - A 390x664 viewport over a portrait stream put it in a left strip with no overlap.
  - Dragging it across moved it to a right strip, sent no input to the page, and was remembered after a reload.
  - A frame exactly the stream's size shrank the video by 5.7% for a bottom band, and a taller frame got a bottom band with no shrink.
  - A 195x422 frame used the overlay handle. Dragged, it snapped to the right edge without sending input. In a 480x270 frame, also an overlay handle, the first tap on the faded handle only woke it and sent nothing to the page, and the second toggled the keyboard. Before, the first tap clicked the page under it.
- Long-press then release produced a contextmenu on the remote page, and long-press then drag across a paragraph selected text.
- Input.insertText of "Hello World" after tapping a textarea arrived as "Hello World".
- The iOS soft keyboard event sequence (a synthetic Shift keydown before the letter, a synthetic CapsLock keydown and keyup around capitals), replayed into the client with navigator.platform set to iPhone so the Guacamole keyboard ran with its iOS quirks, typed "Hello ABc" into Google's search box. That held whether the letter events reported no modifiers, CapsLock or Shift, and with the remote's Caps Lock left on beforehand (it was off afterwards). The previous version typed each letter twice, in the wrong case when the letters reported CapsLock or Shift, and toggled the remote's Caps Lock.
- An earlier version of the keyboard path was also run in Playwright WebKit with an iPhone profile, with the video tracks disabled because Linux WebKit crashes decoding them. The data channel, cursor images, touch input and focus all ran for real, and typed text reached the remote field. Linux WebKit does not use the iOS keyboard code, so this cannot show whether the iOS keyboard opens.
- With the current client, the same swipe only selected text, the keyboard never opened from a tap, input-event text was dropped, and capitals arrived in lower case.
- Desktop mouse click, Shift-typing and the keyboard path were unchanged.
- Keyboard, Copy and Paste on a 390x844 session with iPhone 13 emulation:
  - Tapping plain paragraph text showed no button on the page.
  - Enter in a textarea typed a newline and kept the keyboard open. Enter in a form field that replaces the page closed the keyboard about a second later.
  - Tapping the second field while the keyboard was open on the first kept it open.
  - A long-press drag across a paragraph selected text on the remote page and showed Copy, which put the selection on the local clipboard and showed "Copied". Copy with nothing new copied showed "Nothing was copied".
  - The keys row sat on the visible bottom. Ctrl latched, Ctrl then "a" selected the textarea's text, and the latch cleared. Right then Left moved the caret, and Tab moved focus to the next field.
  - Paste typed a 6-digit code. A two-line, 61-character text was pasted at the caret with only Control and v reaching the page.
  - With clipboard permissions denied, Paste opened the clipboard panel, and text entered there reached the field with "Paste into page".
  - With the frame shortened to 380px and then 200px, the video kept its 390x664 size and moved up 18px and then 198px, keeping the tap above the cut. It refitted at full height.
On a physical iPhone (iOS Safari), an earlier head was tested by hand: scroll and fling, pinch, tap, the keyboard opening on fields, typing including capitals and Caps Lock, dictation, the field staying above the keyboard, Paste with the iOS confirmation, long-press drag selection, dropdowns, and rotation.

Not verified yet:

Not on a physical device since the latest changes (no on-page buttons, closing after Enter, Copy). Not on Android at all. Also unverified: whether iOS shows the form accessory bar while the overlay is focused with inputmode="none", real Gboard behaviour, and timing over cellular networks.
The legacy (non-module) bundle was not exercised.

Known gaps

Chromium shows the text cursor over any selectable text, not only inputs, so in auto mode tapping plain paragraph text can open the keyboard. ?tapKeyboard=off leaves the keyboard to the button.
The keyboard closes after Enter only when the cursor under the last tap leaves the I-beam. A new page with selectable text under that spot keeps it open until the user taps a non-text area or "Hide keyboard".
On iOS, a tap whose cursor is not known yet focuses the overlay for up to 800ms with no keyboard. A hardware keyboard on an iPad types into the remote page during that time, as it would with the keyboard open.
The remote cursor is still drawn in the video. neko's legacy endpoint, which this client uses, selects the legacy stream, captured with show-pointer=true, so the client cannot hide it. Hiding it on touch needs a neko change.

Future work

Native touch input (forwarding real multi-touch to the remote browser) needs no new neko messages: the pinned neko already has touch opcodes (0x08–0x0a), and kernel-images already configures the xf86-input-neko touch driver. It should come with a consistent mobile device profile, since the remote browser currently reports maxTouchPoints 0 and a fine pointer.
A guest-side signal for focused editable fields would replace the cursor heuristic and fix the keyboard staying open after submit.

Note

Medium Risk
Large client-side change to remote input, clipboard, and WebRTC data handling on touch devices; desktop paths are gated but regressions in hosting/control or clipboard sync are possible.

Overview
Adds a touch-first interaction layer to the chromium-headful live view client so phones and tablets can scroll, zoom, type, and use clipboard without treating every swipe as a mouse drag.

Gestures and zoom: Touch handling is replaced with a gesture recognizer (tap, long-press right-click/drag, one-finger scroll with fling, pinch zoom). The stream can be pinch-zoomed locally with coordinate mapping so remote clicks stay accurate; layout uses 100dvh, viewport-fit=cover, and relaxed min-width for narrow embeds.

Keyboard and text: Soft keyboard opens from remote cursor shape (I-beam detection over the data channel) with iOS-specific focus/inputmode timing (tap-keyboard). Android/iOS text goes through a new TextInput path (composition, Shift for capitals, Caps Lock suppressed on touch). A floating keyboard/copy control and an extra keys row (Esc, arrows, Paste, latched Ctrl) sit above the keyboard; the video pans up when the keyboard covers it (visualViewport).

Clipboard: Touch Copy/Paste works within Safari gesture rules (pending ClipboardItem writes, remote clipboard sync); the clipboard panel gets a touch layout with “Paste into page”. The client emits cursor and clipboard events and sets the data channel to arraybuffer for cursor images.

Desktop mouse/keyboard paths are largely unchanged; new logic gates on touch detection and touch events.

^{Reviewed by Cursor Bugbot for commit 8903604. Bugbot is set up for automated code reviews on this repo. Configure here.}

Add a gesture layer for touch input: tap clicks, one-finger swipes scroll with momentum, long-press right-clicks or drags, and two-finger pinch zooms and pans the video locally. Raise the soft keyboard from the tap when the remote cursor is a text cursor, type Android input and composition events, and send Shift for upper case letters from soft keyboards. Replace the small keyboard icon with a labeled button inside the safe area and fix the layout below 360px and under mobile browser toolbars.

cursor

Stale Bugbot comment from a previous run.

…sor shapes Lift the keyboard button by the height the visual viewport loses to an overlaid soft keyboard, and only apply the classification of the newest cursor image.

cursor

Stale Bugbot comment from a previous run.

The keyboard button is positioned inside the player, so lift it by the part of the player the visual viewport no longer shows.

Replace the fixed keyboard button with a draggable control that sits in a band outside the video: below it when there is vertical room, in a strip beside it when there is horizontal room, shrinking the video by up to 10% to make one. Only when there is no room does it overlay the stream as an edge-snapping handle that lets taps through while faded. The control fades when idle in every mode and its position is remembered.

cursor

Stale Bugbot comment from a previous run.

Dragging kept a 44px assumption, so the wider bottom-band control jumped under the finger and landed away from the drop point. Measure the control, keep the finger's offset while dragging, and keep it out of the top and bottom safe areas.

Over a real network the cursor update for a tap usually arrives after the finger lifts, so the keyboard could only be offered through the chip. Track where the current cursor shape was seen and decide inside the tap when it lands on that spot again, such as a second tap on the same field.

cursor

Stale Bugbot comment from a previous run.

A cursor image could be attributed to a newer pointer position while an update for an earlier one was still in flight, and the wait timeout used the latest pointer instead of the tap. Attribute images only when no earlier update can be pending, and pin a timed-out tap to its own point while the pointer is still there.

cursor

Stale Bugbot comment from a previous run.

Pages often show another cursor before the I-beam after a click, and the first update after a tap was taken as the answer, so taps on such fields never offered the keyboard. Any text cursor within the window now offers it, only the end of the window means the tap was not on text, and the same-spot shortcut remembers where a text cursor was seen.

cursor

Stale Bugbot comment from a previous run.

Neko sends no cursor image when the shape does not change, so a tap on another text field while the I-beam was showing ended the wait with no update and dismissed the keyboard. Treat an unchanged I-beam at the end of the wait as text.

pedro-pscunha · 2026-10-10T13:58:38Z

We ran into this with a real user on an iPhone (iOS Safari), on the live view that production serves today (bundle app.416afd11.js, which still has openMobileKeyboard and no tapKeyboard).

Tapping a field in the stream focuses the remote field, but the iOS keyboard never opens, and the user could not find any way to type. The small keyboard icon at the bottom-right does open it. In an emulated iPhone (Playwright WebKit, iPhone 15 Pro) the text then arrived wrong: "Hello" came through as "hello", "!" was lost, and text from autocorrect or predictive suggestions (input events only) was dropped. On the default 1920x1080 window the stream is about 393x221 px on a phone, so the fields are hard to hit too.

CDP Input.insertText with the same kind of string ("S3nh@ Ção") arrived exactly, so the loss is in the client, as this PR describes.

Our case: an agent drives the browser, and a person on a phone types a login and a 2FA code in the live view. This PR (tap-to-type, input/composition handling, Shift for capitals, the 16 px font) covers what we hit. Happy to test a build on a physical iPhone if that helps.

iOS only shows the keyboard for focus() inside a gesture, and over a real network the I-beam usually arrives after the finger lifts, so auto mode fell back to the Tap to type button. Focus the overlay inside the tap with inputmode=none and remove inputmode when a text cursor arrives, which WebKit treats as a request to show the keyboard. Other browsers focus late instead. A tap under an I-beam that is already showing now raises the keyboard at the end of the window, the remembered text spot is measured in screen pixels, and the button is kept for a text cursor that arrives after the window.

cursor

Stale Bugbot comment from a previous run.

A text cursor or the end of the window for an earlier tap could raise the keyboard during a scroll, pinch or drag that started before the cursor arrived. A new touch now cancels the wait and releases a held focus.

masnwilliams · 2026-10-13T19:19:58Z

Thanks for the detailed report @pedro-pscunha — and for offering to test on a physical iPhone.

Good news: we've already validated the keyboard fix on a real iPhone against this build. Tapping a field now raises the iOS keyboard in auto mode (previously it fell through to the "Tap to type" chip), capitals and symbols come through correctly, and tapping non-text dismisses the keyboard. Your login + 2FA case is exactly what this was tested against.

The PR is in review now. Appreciate the offer to test — if anything looks off once this ships, we'd love to hear it.

Sayan-

direction makes sense to me. one question on the protocol choice, and a few ux quirks worth checking.

questions

i see the client-only scope and the concern about consistent browser touch support. did you evaluate migrating the legacy input path to neko’s existing v3 touch messages? the pinned server already handles touch begin/update/end. client, server touch handlers

ux quirks

while zoomed, double-tapping to reset sends the first tap to the remote page, potentially activating a link underneath. is that intended? code
re-enabling input before scroll momentum ends resumes the old swipe without another touch. should disabling input end the gesture? code
on devices classified as touch/coarse-pointer, the added Shift can turn physical-keyboard letters lowercase when Caps Lock is already active remotely. should this account for Caps Lock? code
in Chromium, clicking “Hide keyboard” with a mouse/trackpad blurs the input before the toggle runs, so it focuses the input again. finger taps avoid this. code
cast=1 hides the other controls but can still show “Tap to type.” should it hide that prompt too? code
when the keyboard overlays the view, the new pan limits can leave a bottom-fixed field hidden even after panning fully upward. should they account for the covered area? this is an existing keyboard-visibility gap the new panning does not resolve. code
the faded overlay button intentionally lets taps through, but someone trying to open the keyboard can activate a link underneath. could the first tap wake the button without clicking the page? code

khoi-iso · 2026-10-14T18:50:12Z

Hi! I am very excited about this change and tested it on a physical iPhone (iOS 27 Safari. Thank you for this, it's a big step up.

Works: tapping a text field opened the iOS keyboard on the first tap, and typed text reached the remote field.

Three UX I hit:

The video shrinks when the keyboard opens. The view fits the stream into the smaller visible area, so the page gets narrow, about 60% of the width on our phone.
No way to paste. Long-press becomes a remote right-click, so iOS never shows its own Paste / AutoFill menu. Pasting a password or a 2FA code is one of the most common steps when a person takes over a sign-in on a phone.
The remote mouse cursor is drawn over the page on touch devices. It's confusing next to a finger. Since the client draws the cursor from the cursor images, hiding it when the last input was touch would fix it.

Happy to re-test on device when you push changes.

- Stop a fling whose scroll handler stopped it, and end the page's part of a touch (including a fling) when input stops being enabled, instead of checking for input in every gesture handler. - Decide at touch begin whether the page takes the touch. Without input only a pinch zooms locally. Drop double-tap to reset the zoom, so two quick taps while zoomed are two clicks; the Reset chip stays. - Let a faded keyboard handle over the stream take its own touches; the first one only wakes it. A mouse click on it no longer blurs first. - Move the tap keyboard decisions into a state machine in utils/tap-keyboard.ts with an injected clock, and drop the pointer timing heuristic for the remembered text spot. - Send key events and soft keyboard text through one TextInput path with a single Shift decision that respects Caps Lock, and replace the 50ms dedupe windows with flags. - Hide the Tap to type chip with the other controls, read cursor images only on touch devices, and keep zoom pan limits out of the area an overlaid keyboard covers.

cursor

Stale Bugbot comment from a previous run.

Losing input mid-drag ended the gesture without onDragEnd, so the remote button pressed at drag start was never released.

iOS reports its soft keyboard's Shift and Caps Lock as key events of their own, and the Guacamole keyboard releases every key on iOS right after pressing it. The local Caps Lock check skipped Shift for letters typed after a soft Shift, the soft Caps Lock toggled the remote's Caps Lock, and the immediate release cleared the dedupe flag before the letter's input event, so the letter could be typed twice. On touch devices, never send Caps Lock, and send Shift around a letter when the Shift held on the remote (tracked from the keys sent) and the remote's last known Caps Lock would give it the other case. Only a native keyup ends a key's claim on the next input event, and modifier keys make no claim.

A touch device cannot see the remote's Caps Lock, and touch input never toggles it, so a Caps Lock left on by an earlier session inverted every letter. Turn it off when the keyboard opens and decide each letter's case with Shift alone; the other lock keys are left as they are.

…and park the pointer after touch - While the soft keyboard is open, a frame that only gets shorter keeps the video's size; the video moves up so the last tap stays above the keyboard, and refits when the keyboard closes. - Add a Paste button to the keyboard control while the keyboard is open. It reads the phone clipboard inside the tap and types the text through the soft keyboard path, with a notice when access is denied or unavailable. A native paste on a touch device is typed the same way, and the overlay offers one-time-code AutoFill. - The remote pointer is drawn into the video, so after touch input goes idle move it to the bottom right corner, where it is almost entirely off screen. Mouse input brings it back.

The on-screen position moves during the pan's transition, so recomputing from it on each viewport event drifted past the pan the last tap needed.

cursor

Stale Bugbot comment from a previous run.

…or images in order A tap that moves the pointer while the previous tap's cursor is still pending can receive that tap's I-beam. Treat a text cursor in that window as unsettled: it no longer raises the keyboard on its own, and if no later cursor replaces it by the end of the wait, offer the Tap to type button. Safari delivers data channel messages as Blobs by default, and converting them asynchronously could apply an older cursor image after a newer one. Ask for ArrayBuffers instead.

cursor

Stale Bugbot comment from a previous run.

A non-text tap left the keyboard state in late, so the next field tap treated its own I-beam as possibly stale and showed Tap to type. Only a tap whose wait is still running can have a cursor in flight, and a stale mark now applies only to the touch it arrived during.

…he pointer, and add Copy - Remove the Tap to type button and the chip mode. If the cursor guess misses a field, the keyboard button opens the keyboard. - After Enter with the keyboard open on an I-beam, close the keyboard if the cursor under the last tap is no longer an I-beam a second later (a submitted form or a new page); a chat box keeps its I-beam and the keyboard. - Leave the remote pointer where the touch left it. The legacy stream draws the pointer into the video, so the client cannot hide it. - Add a Copy button to the keyboard control. It starts a clipboard write inside the tap with the text pending, sends Ctrl+C, and fills in the text from the next remote clipboard update, with a notice when nothing was copied or the write is refused.

cursor

Stale Bugbot comment from a previous run.

The remote clipboard update that fills in the Copy control's text also triggered the automatic clipboard sync, whose writeText could replace the pending write and fails outside a tap on iOS. Skip the sync while a Copy write is pending.

… after a selection, and fall back to the clipboard panel - While the soft keyboard is open, a row above it has Esc, Tab, a one-shot Ctrl that stays latched until the next key, the arrows, and Paste. Paste leaves the side controls. - Paste types text up to 32 characters, which also works in fields that block pasting and in one-box-per-character code fields. Longer text is set as the remote clipboard and pasted with Ctrl+V once neko echoes the clipboard back, and typed if it does not. - Copy shows on the side controls for a while after a long-press drag, instead of all the time, and a tap withdraws it. - On touch devices neko's clipboard panel opens when the phone clipboard cannot be read or written, at the top of the view with Paste into page and Close buttons.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5d6e96e. Configure here.}

The panel sits at the top of the player, which is moved up while the keyboard is open, so it could open above the screen.

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue Outdated

Comment thread images/chromium-headful/client/src/components/video.vue

Keep the keyboard button above the soft keyboard and ignore stale cur…

52d6c14

…sor shapes Lift the keyboard button by the height the visual viewport loses to an overlaid soft keyboard, and only apply the classification of the newest cursor image.

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue

masnwilliams added 2 commits October 9, 2026 21:16

Measure the keyboard inset against the player box

facde55

The keyboard button is positioned inside the player, so lift it by the part of the player the visual viewport no longer shows.

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/touch-controls.vue Outdated

Comment thread images/chromium-headful/client/src/components/touch-controls.vue Outdated

masnwilliams added 2 commits October 9, 2026 22:48

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue Outdated

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue Outdated

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue

cursor Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue Outdated

Drop a pending tap keyboard decision when a new touch begins

f4fc300

A text cursor or the end of the window for an earlier tap could raise the keyboard during a scroll, pinch or drag that started before the cursor arrived. A new touch now cancels the wait and releases a held focus.

masnwilliams requested a review from rgarcia October 13, 2026 17:35

Merge branch 'main' into hypeship/live-view-touch-input

3dc5b39

masnwilliams requested review from Sayan- and removed request for rgarcia October 13, 2026 20:57

Sayan- reviewed Oct 13, 2026

View reviewed changes

cursor Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/utils/touch-gestures.ts

masnwilliams added 2 commits October 14, 2026 19:23

Release the mouse button when a drag is cancelled

faa28e0

Losing input mid-drag ended the gesture without onDragEnd, so the remote button pressed at drag start was never released.

masnwilliams added 4 commits October 14, 2026 20:54

Merge branch 'main' into hypeship/live-view-touch-input

eef3025

Measure the keyboard pan from the player's layout position

a13955c

The on-screen position moves during the pan's transition, so recomputing from it on each viewport event drifted past the pan the last tap needed.

cursor Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/utils/tap-keyboard.ts

Comment thread images/chromium-headful/client/src/neko/index.ts

Comment thread images/chromium-headful/client/src/utils/text-input.ts

cursor Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/utils/tap-keyboard.ts Outdated

masnwilliams added 2 commits October 14, 2026 22:18

cursor Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/video.vue

masnwilliams added 2 commits October 14, 2026 22:38

cursor Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread images/chromium-headful/client/src/components/clipboard.vue

Close the soft keyboard before opening the clipboard panel on touch

8903604

The panel sits at the top of the player, which is moved up while the keyboard is open, so it could open above the screen.

masnwilliams mentioned this pull request Oct 14, 2026

Open the live view's soft keyboard from the remote browser's focus #464

Open

Conversation

masnwilliams commented Oct 9, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Known gaps

Future work

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pedro-pscunha commented Oct 10, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

masnwilliams commented Oct 13, 2026

Uh oh!

Sayan- left a comment

Choose a reason for hiding this comment

questions

ux quirks

Uh oh!

khoi-iso commented Oct 14, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

masnwilliams commented Oct 9, 2026 •

edited by cursor Bot

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading